Separable and integrated heat sinks facilitating cooling multi-compnent electronic assembly

ABSTRACT

Cooling apparatuses and methods of fabrication thereof are provided which facilitate cooling a multi-component assembly, such as a hub module assembly. The cooling apparatus includes a first liquid-cooled heat sink configured to facilitate removal of heat generated by one or more first electronic components of the multi-component assembly, and a second liquid-cooled heat sink configured to facilitate removal of heat generated by one or more second electronic components of the multi-component assembly. The first liquid-cooled heat sink is separably coupled to the multi-component assembly, and the second liquid-cooled heat sink is fixedly secured to the multi-component assembly. Fluid couplers fluidically couple the first and second liquid-cooled heat sinks to facilitate liquid coolant flow through the fixedly-secured, second liquid-cooled heat sink from the separably-coupled, first liquid-cooled heat sink.

BACKGROUND

As is known, operating electronic components produce heat. This heatshould be removed in order to maintain device junction temperatureswithin desirable limits, with failure to remove heat effectivelyresulting in increased component temperatures, potentially leading tothermal runaway conditions. Several trends in the electronics industryhave combined to increase the importance of thermal management,including heat removal for electronic components, including technologieswhere thermal management has traditionally been less of a concern, suchas CMOS. In particular, the need for faster and more densely packedcircuits has had a direct impact on the importance of thermalmanagement. First, power dissipation, and therefore heat production,increases as device operating frequencies increase. Second, increasedoperating frequencies may be possible at lower device junctiontemperatures. Further, as more and more devices or components are packedonto a single chip, heat flux (Watts/cm²) increases, resulting in theneed to remove more power from a given size chip or module. These trendshave combined to create applications where it is no longer desirable toremove heat from modern devices solely by traditional air coolingmethods, such as by using air cooled heat sinks with heat pipes or vaporchambers. Such air cooling techniques are inherently limited in theirability to extract heat from an electronic component with high powerdensity. The need to cool current and future high heat load, high heatflux electronic devices therefore mandates the development of aggressivethermal management techniques, using, for instance, liquid cooling.

As an example, some existing supercomputers have compute nodes thatroute their traffic through racks of switching equipment to othercompute nodes. Every switch in this data path adds latency. At asupercomputing scale, there is a point that increasing the number ofcompute drawers will not increase performance due to the additionalswitching latency.

In a system using hub modules, networking and compute traffic is routedto idle compute processors with the hub modules to maximize speed andefficiency. In the system, every compute drawer is directly connected toevery other compute drawer via the hub modules, which typically includetraffic routing hub chips and a network of fiber-optic transmit andreceive modules.

In a system with a network of fiber optic transmit and receive modulesor fiber optic interconnects, scalability is enabled to a much higherlevel than previously possible. However, a problem exists creating areliable arrangement having manufacturability and delivering a requiredpackage density and heat removal.

BRIEF SUMMARY

The shortcomings of the prior art are overcome and additional advantagesare provided through, in one aspect, the provision of a coolingapparatus which includes a first liquid-cooled heat sink, a secondliquid-cooled heat sink, and fluid couplers fluidically coupling thefirst and second liquid-cooled heat sinks The first liquid-cooled heatsink, which includes at least one coolant-carrying first channel, isseparably coupled to an electronic assembly comprising at least onefirst electronic component and at least one second electronic component,and facilitates removal of heat generated by the at least one firstelectronic component. The second liquid-cooled heat sink includes atleast one coolant-carrying second channel, and is fixedly secured to theelectronic assembly to facilitate removal of heat generated by the atleast one second electronic component. The fluid couplers fluidicallycouple the first and second liquid-cooled heat sinks together andfacilitate liquid coolant flow through the at least one coolant-carryingsecond channel of the fixedly-secured, second liquid-cooled heat sinkfrom the separably-coupled, first liquid-cooled heat sink.

In another aspect, a cooled electronic system is provided which includesan electronic assembly, and a cooling apparatus. The electronic assemblyincludes at least one first electronic component and at least one secondelectronic component, and the cooling apparatus is coupled to theelectronic assembly. The cooling apparatus includes a firstliquid-cooled heat sink, a second liquid-cooled heat sink, and fluidcouplers. The first liquid-cooled heat sink, which includes at least onecoolant-carrying first channel, is separably coupled to the electronicassembly and facilitates removal of heat generated by the at least onefirst electronic component. The second liquid-cooled heat sink includesat least one coolant-carrying second channel, and is fixedly secured tothe electronic assembly to facilitate removal of heat generated by theat least one second electronic component. The fluid couplers fluidicallycouple the first and second liquid-cooled heat sinks and facilitateliquid coolant flow through the at least one coolant-carrying channel ofthe fixedly-secured, second liquid-cooled heat sink from theseparably-coupled, first liquid-cooled heat sink.

In a further aspect, a method is provided which includes fabricating acooling apparatus to facilitate cooling an electronic assemblycomprising at least one first electronic component and at least onesecond electronic component. Fabricating the cooling apparatus includes:separably coupling a first liquid-cooled heat sink to the electronicassembly to facilitate removal of heat generated by the at least onefirst electronic component, the first liquid-cooled heat sink comprisingat least one coolant-carrying first channel; fixedly securing a secondliquid-cooled heat sink to the electronic assembly to facilitate removalof heat generated by the at least one second electronic component, thesecond liquid-cooled heat sink comprising at least one coolant-carryingsecond channel; and providing fluid couplers fluidically coupling thefirst and second liquid-cooled heat sinks, the fluidic couplingfacilitating liquid coolant flow through the at least onecoolant-carrying second channel of the fixedly-secured, secondliquid-cooled heat sink from the separably-coupled, first liquid-cooledheat sink.

Additional features and advantages are realized through the techniquesof the present invention. Other embodiments and aspects of the inventionare described in detail herein and are considered a part of the claimedinvention.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

One or more aspects of the present invention are particularly pointedout and distinctly claimed as examples in the claims at the conclusionof the specification. The foregoing and other objects, features, andadvantages of the invention are apparent from the following detaileddescription taken in conjunction with the accompanying drawings inwhich:

FIG. 1 depicts one embodiment of a conventional raised floor layout ofan air-cooled data center;

FIG. 2 depicts one embodiment of an at least partially liquid-cooleddata center which includes a coolant distribution unit facilitatingliquid-cooling of electronics racks of the data center, in accordancewith one or more aspects of the present invention;

FIG. 3 depicts a cross-sectional elevational view of one embodiment of acooled electronic system comprising a hub module assembly and oneembodiment of a cooling apparatus, in accordance with one or moreaspects of the present invention;

FIG. 4A depicts an isometric view of the cooled electronic system ofFIG. 3, without the loading and cold plate assembly thereof, inaccordance with one or more aspects of the present invention;

FIG. 4B is a side elevational view of the partial cooled electronicsystem of FIG. 4A, in accordance with one or more aspects of the presentinvention;

FIG. 4C is an exploded perspective view of the cooled electronic systemof FIGS. 4A & 4B, in accordance with one or more aspects of the presentinvention;

FIG. 5 is an exploded perspective view of one embodiment of a loadingassembly of the cooled electronic system of FIG. 3, in accordance withone or more aspects of the present invention;

FIG. 6 depicts one embodiment of multiple liquid-cooled heat sinks andloading assemblies coupled in series-fluid communication forfacilitating cooling of multiple electronic assemblies, such as multipleones of the hub module assembly depicted in FIGS. 3-5, in accordancewith an aspect of the present invention;

FIG. 7A is an isometric view of another embodiment of a cooledelectronic system comprising a hub module assembly and an alternateembodiment of a cooling apparatus, in accordance with one or moreaspects of the present invention;

FIG. 7B is a partially exploded view of the cooled electronic system ofFIG. 7A, with the separably-coupled, first liquid-cooled heat sink shownexploded from the cooled electronic system and illustrating removal ofthe fluid couplers from respective fluid ports of the fixedly-secured,second liquid-cooled heat sink of the cooling apparatus, in accordancewith one or more aspects of the present invention;

FIG. 8A is an isometric view of one embodiment of the fixedly-secured,second liquid-cooled heat sink of the cooling apparatus of FIGS. 7A &7B, in accordance with one or more aspects of the present invention;

FIG. 8B is a partially exploded view of the fixedly-secured, secondliquid-cooled heat sink of FIG. 8A, in accordance with one or moreaspects of the present invention;

FIG. 9A is a top isometric view of one embodiment of theseparably-coupled, first liquid-cooled heat sink of FIGS. 7A & 7B, inaccordance with one or more aspects of the present invention;

FIG. 9B is a bottom isometric view of the separably-coupled, firstliquid-cooled heat sink of FIG. 9A, in accordance with one or moreaspects of the present invention;

FIG. 9C is a partially exploded view of the separably-coupled, firstliquid-cooled heat sink of FIGS. 9A & 9B, in accordance with one or moreaspects of the present invention; and

FIG. 9D is a top plan view of the cooling apparatus of FIGS. 7A-9C, andillustrating a liquid coolant flow path through the separably-coupled,first liquid-cooled heat sink, and then the fixedly-secured, secondliquid-cooled heat sink, with the first and second liquid-cooled heatsinks being operatively fluidically coupled via the fluid couplers, inaccordance with one or more aspects of the present invention.

DETAILED DESCRIPTION

Aspects of the present invention and certain features, advantages, anddetails thereof, are explained more fully below with reference to thenon-limiting embodiments illustrated in the accompanying drawings.Descriptions of well-known materials, fabrication tools, processingtechniques, etc., are omitted so as not to unnecessarily obscure theinvention in detail. It should be understood, however, that the detaileddescription and the specific examples, while indicating embodiments ofthe invention, are given by way of illustration only, and unlessotherwise specified, are not by way of limitation. Varioussubstitutions, modifications, additions and/or arrangements within thespirit and/or scope of the underlying inventive concepts will beapparent to those skilled in the art from this disclosure.

As used herein, the terms “electronics rack” and “rack unit” are usedinterchangeably, and unless otherwise specified include any housing,frame, rack, compartment, blade server system, etc., having one or moreheat-generating components of a computer system, electronic system, orinformation technology equipment, and may be, for example, a stand-alonecomputer processor having high-, mid- or low-end processing capability.In one embodiment, an electronics rack may comprise a portion of anelectronic system, a single electronic system, or multiple electronicsystems, for example, in one or more sub-housings, blades, books,drawers, nodes, compartments, etc., having one or more heat-generatingelectronic components disposed therein. An electronic system(s) withinan electronics rack may be movable or fixed relative to the electronicsrack, with rack-mounted electronic drawers and blades of a blade-centersystem being two examples of electronic systems (e.g., subsystems ornodes) of an electronics rack to be cooled.

“Electronic component” refers to any heat generating electroniccomponent of, for example, an electronic system or other unit requiringcooling. By way of example, an electronic component may comprise one ormore integrated circuit dies and/or other electronic devices to becooled, including one or more processor dies, memory dies or memorysupport dies. As a further example, the electronic component maycomprise one or more bare dies or one or more packaged dies disposed ona common carrier. Further, unless otherwise specified herein, the terms“liquid-cooled heat sink”, or “liquid-cooled structure” each refer toany conventional thermally conductive structure having a plurality ofchannels or passageways formed therein for flowing of liquid-coolanttherethrough.

As used herein, a “liquid-to-liquid heat exchanger” may comprise, forexample, two or more coolant flow paths, formed of thermally conductivetubing (such as copper or other tubing) in thermal or mechanical contactwith each other. Size, configuration and construction of theliquid-to-liquid heat exchanger can vary without departing from thescope of the invention disclosed herein. Further, “data center” refersto a computer installation containing, for example, one or moreelectronics racks to be cooled. As a specific example, a data center mayinclude one or more rows of rack-mounted computing units, such as serverunits.

One example of facility coolant and system coolant is water. However,the concepts disclosed herein are readily adapted to use with othertypes of coolant on the facility side and/or on the system side. Forexample, one or more of these coolants may comprise a brine, adielectric liquid, a fluorocarbon liquid, a liquid metal, or othersimilar coolant, or refrigerant, while still maintaining the advantagesand unique features of the present invention.

Reference is made below to the drawings, which are not drawn to scalefor ease of understanding, wherein the same reference numbers usedthroughout different figures designate the same or similar components.

FIG. 1 depicts a raised floor layout of an air cooled data center 100typical in the prior art, wherein multiple electronics racks 110 aredisposed in one or more rows. A data center such as depicted in FIG. 1may house several hundred, or even several thousand microprocessors. Inthe arrangement illustrated, chilled air enters the computer room viaperforated floor tiles 160 from a supply air plenum 145 defined betweenthe raised floor 140 and a base or sub-floor 165 of the room. Cooled airis taken in through louvered covers at air inlet sides 120 of theelectronics racks and expelled through the backs, that is, air outletsides 130, of the electronics racks. Each electronics rack 110 may haveone or more air moving devices (e.g., fans or blowers) to provide forcedinlet-to-outlet airflow to cool the electronic devices within thesubsystem(s) of the rack. The supply air plenum 145 provides conditionedand cooled air to the air-inlet sides of the electronics racks viaperforated floor tiles 160 disposed in a “cold” aisle of the computerinstallation. The conditioned and cooled air is supplied to plenum 145by one or more air conditioning units 150, also disposed within datacenter 100. Room air is taken into each air conditioning unit 150 nearan upper portion thereof. This room air may comprise in part exhaustedair from the “hot” aisles of the computer installation defined, forexample, by opposing air outlet sides 130 of electronics racks 110.

Due to the ever-increasing airflow requirements through electronicsracks, and the limits of air distribution within the typical data centerinstallation, liquid-based cooling is being combined with theabove-described conventional air-cooling. FIGS. 2-3 illustrate oneembodiment of a data center implementation employing a hybrid air- andliquid-based cooling system with one or more liquid-cooled heat sinks orcold plates coupled to high heat-generating electronic componentsdisposed within the electronics racks.

FIG. 2 depicts one embodiment of an at least partially liquid-cooleddata center which includes a coolant distribution unit 200 having apower/control element 212, a reservoir/expansion tank 213, a heatexchanger 214, a pump 215 (possibly accompanied by a redundant secondpump), facility water inlet 216 and outlet 217 supply pipes, a supplymanifold 218 supplying water or system coolant to the electronics racks210 via couplings 220 and lines 222, and a return manifold 219 receivingwater from the electronics racks 210, via lines 223 and couplings 221.Each electronics rack includes (in one example) a power/control unit 230for the electronics rack, multiple electronic systems 240, a systemcoolant supply manifold 250, and a system coolant return manifold 260.In this embodiment, each electronics rack 210 is disposed on raisedfloor 140 of the data center, with lines 222 providing system coolant tosystem coolant supply manifolds 250 and lines 223 facilitating return ofsystem coolant from system coolant return manifolds 260 shown disposedin the supply air plenum beneath the raised floor.

In the embodiment illustrated, the system coolant supply manifold 250provides system coolant to the cooling systems of the electronic systems(more particularly, for example, to liquid-cooled cold plates thereof)via flexible hose connections 251, which are disposed between the supplymanifold and the respective electronic systems within the rack.Similarly, system coolant return manifold 260 is coupled to theelectronic subsystems via flexible hose connections 261. Quick connectcouplings may be employed at the interface between flexible hoses 251,261 and the individual electronic systems. By way of example, thesequick connect couplings may comprise various types of commerciallyavailable couplings, such as those available from Colder ProductsCompany, of St. Paul, Minn., USA, or Parker Hannifin, of Cleveland,Ohio, USA.

Although not shown, electronics rack 210 may also include anair-to-liquid heat exchanger disposed at an air outlet side thereof,which also receives system coolant from the system coolant supplymanifold 250 and returns system coolant to the system coolant returnmanifold 260.

Within the electronics racks, system coolant may be provided to avariety of cooled electronic assemblies, including, for instance, to aliquid-cooled hub module assembly, such as described below withreference to FIGS. 3-5.

FIG. 3 depicts a cooled electronic assembly, generally designated 300,in accordance with one or more aspects of the present invention. In thisexample, the cooled electronic assembly is a hub module assembly whichincludes a hub module components assembly generally, designated by thereference character 301, and a cold plate assembly, generally designated302. Cooled electronic assembly 300 includes a cooling apparatusimplementing enhanced loading and heat removal for the self-containedunitary hub module assembly.

The hub module assembly includes a hub chip 304 carried by a hub ceramicsubstrate 306, and a plurality of optical modules 308 attached by a topsurface metallurgy (TSM) land grid array (LGA) assembly 310 residing onthe hub ceramic substrate 306. The ceramic substrate 306 is connected toa circuit board 312 through a bottom surface metallurgy (BSM) LGAinterposer 314. The circuit card 312 is mounted on an associatedbackside stiffener member 316 separated by an insulator 318.

Referring also to FIGS. 4A-4C, hub module components 301 are shown indetail. A central hub chip heat spreader 320 (such as a copper block)resides on the hub chip 304 shown in FIG. 3. A pair of bonded mountingangle brackets 322 are secured to opposed long sides 324 of the ceramicsubstrate 306. A unitary base alignment ring 326 includes a plurality ofalignment features including alignment dowel pins 328 received throughLGA interposer alignment aperture 330 with retention features or nubsfor engaging the circuit board 312 shown in FIG. 3 and locating the BSMLGA interposer 314 with the circuit board. The unitary base alignmentring 326 is optically aligned with the BSM LGA pad array on the ceramicsubstrate 306. Each of a pair of top alignment rings 332 includescooperating alignment features 334, such as dowel pins 334 receivedwithin a respective LGA interposer alignment aperture 336 with retentionfeatures or nubs for engaging and locating respective LGA sites of arespective LGA interposer 340 of the TSM LGA assembly 310. The pair oftop alignment rings 332 are optically aligned with the respective TSMLGA pad array on the ceramic substrate 306. The two LGA interposers 340of the TSM LGA assembly 310 align, retain, and make the electricalconnection between the optical modules 308 and the hub chip 304.

Each of the two LGA interposers 340 includes molded features 341 in theinterposer that act as springs to center the optical modules 308. Also,features in the alignment apertures 336 center the holes on the pins aswell as aid retention of a loose interposer onto the dowel pins. Themolded features 341 include small hook features which retain individualmodules 308 once they are set on the LGA interposer 340.

A plurality of lower alignment ring mounting fasteners 342 are receivedthrough corresponding respective apertures 344 having requiredpositioning clearance for the optical alignment process and threadedapertures 346 in the unitary base alignment ring 326 and the edge bondedmounting angle brackets 324. A plurality of upper alignment ringmounting fasteners 348 are received through corresponding respectiveapertures 350 having required positioning clearance for the opticalalignment process and threaded apertures 352 in the pair of topalignment rings 332 and the unitary base alignment ring 326.

The base alignment ring 326 and the pair of top alignment rings 332 areattached to the mounting brackets 322. A heat removal and load assembly354 includes a respective global heat spreader member 356 provided for arespective group of the plurality of optical modules 308 to remove heatand apply module load at the respective LGA sites on the top surfacemetallurgy (TSM) LGA assembly 310.

By way of example, the hub module assembly may include 56 opticalmodules 308 arranged in two groups of 28 optical modules 308 mirroredabout the center hub chip 304. The optical modules 308 may be opticallaser transmitter and receiver modules having via land grid array (LGA)connections on the top surface metallurgy (TSM) LGA assembly 310residing on the ceramic substrate 306.

The heat removal and load assembly 354 includes the global heat spreader356 shown in FIG. 4C in contact with each of the plurality of opticalmodules 308 to facilitate loading of the modules to make the LGAinterconnect, and also to remove heat created during operation. Eachmodule 308 has, in one embodiment, a copper saddle 360 best shown inFIG. 4C, that is loaded against the global heat spreader 356 through thefirst thermal interface material 362 in the heat removal path. Eachcopper saddle 360 has a small coil spring 364 bearing down on it toprovide the controlled load required to make a reliable LGA connection,and to protect from overloading individual optical modules 308.

In accordance with aspects of the present invention, due to the highload required on the overall assembly, for example, approximately 680lbs and the small load required on each optical module, such as lessthan 10 lbs, the coil springs 364 function as buffers, preventing thetallest optical module 308 from being crushed and the shortest fromgetting no load. The coil springs 364 are bonded into cavities in theglobal heat spreader 356. The global heat spreader 356 has, forinstance, a perforated sheet 366 of thermal interface material (TIM)with respective openings 368 on the base that allows each coil spring364 to pass through, but touches the remaining area of the top of eachsaddle 356. The optical modules 308 transmit and receive through a flatfiber ribbon 370 that escapes horizontally.

Due to the hand-plug nature of the hub modules 308 and the potential ofaccidentally influencing the positions of modules 308 while handling andmanipulating fiber 370, for example, due to the light preload, a strainrelief assembly 372 is provided to isolate the optical modules from theterminal ends of the fiber 370.

As shown in FIGS. 3-4C, the strain relief assembly 372 includes a flatribbon strain relief member 374, a strain relief cover member 376, aplurality of strain relief cover mounting screws 378, and a plurality offiber ribbon strain relief mounting fasteners 380. A plurality of globalheat spreader mounting fasteners 382 are received through each of theglobal heat spreader members 356 fastening to the mounting brackets 322.An upper, final thermal interface 384 is provided by a respective thinsheet of indium resting on the top surface of the global heat spreadermembers 356 and the central copper heat spreader 320 residing on the hubchip 304.

Referring also to FIG. 5, there is shown a cold plate assembly generallydesignated 302 of the hub module assembly. The cold plate assembly 302includes a water cooled cold plate 386. Cooling water is circulatedthrough conduits 387. The indium thin sheet final thermal interface 384provided on the top surface of both global heat spreaders 356 and thecentral copper heat spreader 320 is located in contact engagement withthe base of the cold plate 386. The indium pads 384 can be reused if asystem is reworked within manufacturing, which reduces cost anddecreases system rework cycle time.

The cold plate assembly 302 of FIGS. 3 & 5 includes a laminated springplate 388, a centrally located fastener 390, and a pair of spring endsupport brackets 392 receiving spring end support bracket fasteners 394and mounted to a respective card mounted cross brace 396. A pair of longstiffening rails 398 provided between the card mounted cross brace 396and mounted to the circuit board 312. The long stiffening rails 398 andthe cold plate 386 includes respective cooperating alignment features500 and 502 positioning and retaining the cold plate 386.

The cold plate load is provided by fastening the plurality of fasteners394 on the spring end support brackets 392 and the card mounted crossbraces 396, and bottoming the head of the load fastener 390 against thelaminated spring plate 388. A total system load of 680 lbs is generatedby deflecting the laminated spring plate 388 retained at both ends viathe centrally located screw 390. The screw tip acts directly on the coldplate assembly 302, driving the reaction load vertically through thehardware stack.

When the hub module 301 is not loaded by a cold plate 386, the globalheat spreader top surfaces are higher than the top of the central copperheat spreader 320 residing on the hub chip. Each global heat spreader356 is captivated by the global heat spreader mounting shoulderfasteners 382, and in this condition there is some small coil springcompression that maintains a preload on the optical modules 308. Thismaintains optical module position, as well as reducing wear on the goldLGA pads due to vibration induced surface scrubbing. When the cold plateload is applied, both global heat spreaders 356 move down, compressingthe array of coil springs 364, increasing the module load at therespective TSM LGA sites. When the top surfaces of the global heatspreaders 356 and the central copper heat spreader 320 are coplanar, thefull design load has been applied to the optical module TSM LGA sites.The physical down stop of the central copper heat spreader 320 preventsoverloading these components, because the coil springs 364 can no longerbe compressed. Additional loading after the surfaces are coplanar passesthrough the central copper heat spreader 320 and hub chip 304 to the BSMLGA connection onto the circuit board 312, but does not increase theload on the optical modules 308 or TSM 310. By design, and as oneexample only, 200 lbs load passes through each global heat spreader 356to each bank of 28 optics modules 108, and 280 lbs passes through thecenter spreader 320, resulting in a total of 680 lbs nominal on the BSMLGA 314. These loads are defined by how many LGA contacts are present,and the contact force requirement per contact.

In brief, the cooled electronic assembly (e.g., hub module assembly) isa reliable arrangement with effective manufacturability that deliversrequired package density. One of the main principles of the hub moduleassembly is to push complexity of system assembly into the hub modulecomponents assembly 301 by making the module self contained, test-ableand shippable at the unit level, as well as hand place-able. The hubmodule components assembly 301 includes a large number of components,thermal interfaces, and springs while on the system manufacturing floor,the hub module is installed by hand, and the cold plate assembly 302applies the cold plate load provided by fastening the four fasteners 394on the spring end supports 392, 396, and bottoming the load fastener 390against the laminated spring plate 388. Also, due to the water coolednature of the hub module assembly 300, heat is effectively moved to thetop thermal interface 384 of the hub module components assembly 301contacting the single cold plate 386.

As one specific example, the hub module assembly of FIGS. 4-5 maycomprise an IBM® Power® 775™ Supercomputer Input/Output (I/O) hubmodule, offered by International Business Machines Corporation ofArmonk, N.Y., USA. (IBM®, Power®, and Power 775™ are trademarks ofInternational Business Machines Corporation, Armonk, N.Y., USA.) ThisI/O hub module is a complex structure that routes compute trafficbetween a multitude of processors in the compute cluster. Integratingthis function into a central electronic complex (CEC) or centralprocessor complex (CPC) reduces switching latency, and thus improvesparallel processing efficiency and eliminates external switch gear,resulting in a smaller compute footprint on the data center floor. Themodule packages as a “hub” chip and, in one embodiment, 56 individualopto-electrical transmitter/receiver modules (referred to herein as theoptical modules) for electrical interconnection and power dissipation.

By way of specific example, the I/O hub module may have a base that is a95 mm long×61 mm wide×7.5 mm thick glass ceramic substrate, withindividual wiring layers for power and signal distribution. Located atthe center at the top side of the substrate is the hub chip. In oneexample, this chip may be 22.05 mm×26.88 mm (592.8 mm²) in size, andelectrically connected to the substrate by more than 11,000 electricalcontacts, such as flip chip (C4) solder interconnects. Cooling for sucha hub chip is provided by the hub chip heat spreader in the exampleconfiguration of FIGS. 3-5 through, for instance, a thin silicone-based,thermally conductive adhesive between the hub chip and hub chip thermalspreader. This chip thermal interface (TIM1) may have a unit thermalresistance of, for instance 10° C. mm²/W.

As noted, the optical modules (e.g., 56 optical modules) are dividedinto two groups of, for instance, 28 modules each, mirrored about thecenter hub chip. The optical modules have, in one embodiment, copperside blocks to facilitate loading of the modules to make the LGAinterconnect, and also to remove heat created during operation. Eachmodule has, by way of example, a copper saddle that is loaded againstthese blocks through the first thermal interface material in the heatremoval path (i.e., TIM1). Each copper saddle has a small coil springbearing on it to provide the controlled load required to make a reliableLGA connection, and to protect from overloading individual opticalmodules. Due to the high load required on the overall assembly (e.g.,approximately 680 lbf), and the small load required on each opticalmodule (e.g., approximately 7.1 lbf), the coil spring acts as a bufferthat prevents the tallest optical module from being crushed, and theshortest optical module from receiving no load. The coil springs arebonded into cavities in the respective global heat spreader.

When the hub module is not loaded by a cold plate, the global heatspreader top surfaces are higher than the top of the central copper heatspreader residing on the hub chip. In this unloaded position, the globalheat spreader is captivated by two shoulder fasteners. In thiscondition, there is some small coil spring compression that maintains apre-load on the optical modules. This maintains optical module position,as well as reducing wear on the LGA pads due to vibration-inducedsurface scrubbing. When the cold plate load is applied, both global heatspreaders move downward, compressing the array of coil springs,independently loading each optical module and TSM LGA site. At the sametime, the global spreader is compressing a thermal interface pad (TIM3)between it and the copper saddles. When the top surfaces of the globalheat spreaders and the central copper heat spreader are coplanar, thefull design load has been applied at optical module TSM LGA sites viaeach coil spring, the physical down-stop of the central copper heatspreader prevents overloading these components, because the coil springscan no longer be compressed. Additional loading after the surfaces arecoplanar passes through the central copper heat spreader and chip to theBSM LGA connection onto the motherboard, but does not increase the loadon the optical modules or TSM. As noted, by design, 200 lbf load maypass through heat global heat spreader to each back of 28 opticsmodules, and 280 lbf passes through the center spreader, resulting in atotal of 680 lbf nominal on the BSM LGA. The total system load of 680lbf is generated by deflecting a laminated spring plate retained at bothends via a centrally located fastener. The fastener tip acts directly onthe cold plate assembly, driving the reaction load vertically throughthe hardware stack. Both global heat spreaders and the central copperheat spreaders have a thin sheet of, for instance, indium resting on thetop surface, which acts as a final thermal interface (TIM2), which is incontact with the base and the cold plate. Indium is particularlybeneficial in this application, as grease could be problematic in thecomplicated assembly, and exceedingly difficult to rework, given thelack of continuous surface. Indium pads can be reused if a system isreworked within manufacturing, which reduces costs and decreases systemrework cycle time.

As illustrated in FIG. 6, at the CEC level, four hub module cold platesmay be plumbed together in serial flow via appropriate liquid-coolantflow tubing 600 between a coolant supply manifold connection 601 and acoolant return manifold connection 602 as depicted. The temperature ofthe coolant, for instance, water, entering the fourth hub module coldplate 386 in the series increases by the caloric temperature rise due tothe three upstream modules' heat loads. This temperature differentialneeds to be considered when assuring component temperatures are withinspecification.

As a further example, I/O module power dissipation may be approximately340 W, with the hub chip dissipating 107 W, and each optic moduledissipating 2.5 W. Future hub module assembly configurations maydissipate even greater heat load, requiring even greater cooling of thehub chip, while also cooling the optics modules.

Disclosed herein with reference to FIGS. 7A-9D, therefore, is anenhanced cooled electronic assembly or hub module assembly, where aportion of the cooling apparatus (i.e., a second liquid-cooled heatsink) is fixedly secured (e.g., integrated) with the electronicsassembly, and another portion of the cooling apparatus (i.e., a firstliquid-cooled heat sink) is separably coupled to the assembly to makefluid communication with the integrated cooling portion, while alsocooling other portions of the module assembly in the more traditional,separable fashion. For instance, the second liquid-cooled heat sink(e.g., cold plate) is integrated with or bonded directly to at least oneof the electronic components of a heterogeneous, multi-component moduleor assembly (e.g., to the hub chip), and the separable, firstliquid-cooled heat sink (e.g., separate cold plate), when brought intoboth thermal and mechanical contact with the electronic assembly ormodule, makes a fluidic connection to the fixedly-secured, secondliquid-cooled heat sink.

Generally stated, disclosed herein is a cooling apparatus which includesa first liquid-cooled heat sink, a second liquid-cooled heat sink, andfluid couplers. The first liquid-cooled heat sink includes at least onecoolant-carrying first channel, and is separably-coupled to theelectronic assembly to facilitate removal of heat generated by at leastone first electronic component of the electronic assembly. The secondliquid-cooled heat sink includes at least one coolant-carrying secondchannel, and is fixedly-secured to the electronic assembly andconfigured to facilitate removal of heat generated by at least onesecond electronic component of the electronic assembly. The fluidcouplers fluidically couple the first and second liquid-cooled heatsinks, and facilitate liquid coolant flow through the at least onecoolant-carrying second channel of the fixedly-secured, secondliquid-cooled heat sink from the separably-coupled, first liquid-cooledheat sink.

In one implementation, the second liquid-cooled heat sink may beintegrated with the multi-component electronic assembly comprising theat least one first electronic component and the at least one secondelectronic component. The integration may be by, for instance, byadhesively affixing the second liquid-cooled heat sink to the at leastone second electronic component to be cooled.

The fluid couplers may project from either the first liquid-cooled heatsink or the second liquid-cooled heat sink, and in one example, arerigid, cylindrical-shaped projections, each with one or more sealingrings around a periphery thereof. The fluid couplers are sized andconfigured to project into respective coolant ports in the other of thefirst liquid-cooled heat sink or second liquid-cooled heat sink withoperative coupling of the first liquid-cooled heat sink and the secondliquid-cooled heat sink. A loading mechanism, such as described above,applies a compressive loading to the first and second liquid-cooled heatsinks, and this compressive loading facilitates forming a fluid-tightseal between the first and second liquid-cooled heat sinks about thefluid couplers disposed within the respective coolant ports. In analternate embodiment, sealing rings may also be associated with thecoolant ports to facilitate fluid-tight coupling between the two heatsinks with, for instance, compressive loading of the first liquid-cooledheat sink onto the second liquid-cooled heat sink.

In one example, liquid coolant flows through the at least onecoolant-carrying second channel of the second liquid-cooled heat sink,after passing through the at least one coolant-carrying first channel ofthe first liquid-cooled heat sink. The first liquid-cooled heat sink mayfacilitate removal of heat generated by multiple first electroniccomponents, and the at least one second electronic component may residein between different groups of first electronic components, as in thecase described above in connection with the hub module assembly of FIGS.3-5. The second liquid-cooled heat sink is disposed over the at leastone second electronic component, and in thermal communication therewith.The multiple first electronic components may comprise multiple opticalmodules of a hub module assembly, and the at least one second electroniccomponent may comprise the hub chip of the hub module assembly. In thisconfiguration, the first liquid-cooled heat sink extends at leastpartially over the multiple optical modules to facilitate removal ofheat therefrom, and the second liquid-cooled heat sink physicallyengages the hub chip, and is disposed between the hub chip and the firstliquid-cooled heat sink. A first global optic heat spreader and a secondglobal optic heat spreader may be disposed over respective groups ofoptical modules on the opposite sides of the hub chip, and the first andsecond global optic heat spreaders facilitate transfer of heat from themultiple optical modules, to the first liquid-cooled heat sink, as wellas, in one implementation, the second liquid-cooled heat sink, that is,assuming that the global optic heat spreaders are in thermalcommunication with the second liquid-cooled heat sink.

As one example, the first liquid-cooled heat sink comprises a pluralityof coolant-carrying first channels arrayed in parallel, for instance,formed by a plurality of parallel-extending fins, and the secondliquid-cooled heat sink comprises a plurality of coolant-carrying secondchannels arrayed in parallel, and formed, for instance, via a pluralityof parallel-extending fins. In one implementation, the plurality offirst channels of the first liquid-cooled heat sink are orientedorthogonal to the plurality of coolant-carrying second channels of thesecond liquid-cooled heat sink.

By way of specific example, FIGS. 7A & 7B depict one embodiment of acooled electronic system, generally denoted 700, illustrated without theloading mechanism. In one instance, the loading mechanism is similar tothat described above in connection with the cooled electronic assemblyembodiment of FIGS. 3-5. In this example, the cooled electronic systemcomprises a multi-component assembly, such as the above-described hubmodule assembly of FIGS. 3-5. The cooling apparatus of the cooledelectronic assembly comprises a first liquid-cooled heat sink 710 and asecond liquid-cooled heat sink 720. In this example, the secondliquid-cooled heat sink is (for instance) a liquid-cooled cold plateconfigured to reside between the global optic heat spreaders 356 of ahub module assembly, and be in thermal communication with andmechanically coupled to the one or more second electronic components tobe cooled, in this example, the hub chip disposed between the differentgroups of optics modules.

In one example, the first and second liquid-cooled heat sinks 710, 720are each fabricated of metal and each include one or morecoolant-carrying channels extending therethrough. By way of example, theliquid coolant flowing through the first and second liquid-cooled heatsinks may comprise water. In the depicted implementation, the fluidcouplers 711, 712 project from the bottom surface of the firstliquid-cooled heat sink 710, and are sized and configured to fluidicallycouple in a fluid-tight manner to the second liquid-cooled heat sink 720within respective coolant ports 721, 722 (FIG. 7B) of the secondliquid-cooled heat sink 720. Note that the cooled electronic assembly700 of FIGS. 7A & 7B is similar to that described in connection withFIGS. 3-5, except that the solid thermal spreader 320 of FIGS. 3-5 hasbeen replaced with the fixedly-secured, second liquid-cooled heat sink(or cold plate), as described herein, and a mechanism is provided toensure fluid-tight coupling between the first and second liquid-cooledheat sinks In this configuration, the second liquid-cooled heat sink maybe bonded to the electronic assembly substrate, rendering the secondliquid-cooled heat sink fixedly secured to or integral with theelectronic assembly.

FIGS. 8A & 8B depict one embodiment of the fixedly-secured, secondliquid-cooled heat sink 720, illustrating the structure in greaterdetail. In this two-piece configuration, second liquid-cooled heat sink720 includes a lid structure 800 and a base structure 810, which aresized and configured to seal together and define therebetween the one ormore coolant-carrying second channels of the second liquid-cooled heatsink. A plurality of fins 815 are illustrated extending from basestructure 810. The plurality of fins may comprise a plurality ofthermally conductive fins, such as thermally conductive plate finsspaced apart in parallel, opposing relation to define a plurality ofsecond channels extending therethrough. When lid structure 800 is sealedto base structure 810, and liquid coolant is provided via coolant ports721, 722, the liquid coolant flows through the channels defined betweenthe plurality of fins 815 and thereby facilitates removal of heat fromthe second liquid-cooled heat sink. This structure provides the means todeliver liquid coolant, such as water, in close proximity to the one ormore second electronic components of the multi-component assembly to becooled. The lid structure and base structure of the second liquid-cooledheat sink are thermally conductive and may be brazed together to formthe cold plate structure. Note that in this configuration, a single passcross-flow configuration is shown by way of example only. Both the heattransfer enhancement, and the flow configuration through the secondliquid-cooled heat sink may be achieved in a multitude of ways, withoutdeparting from the nature of the inventive concepts disclosed herein.

FIGS. 9A-9C depict one embodiment of first liquid-cooled heat sink 710.In this embodiment, first liquid-cooled heat sink 710 comprises atwo-piece embodiment which includes a lid structure 900 and a basestructure 910, with a plurality of thermally conductive fins 915 shownextending from base structure 910 into an appropriately-sized recess 901in lid structure 900. The plurality of fins 915 comprise, for instance,a plurality of plate-type fins disposed in spaced, opposing relation todefine multiple, parallel single-pass, coolant-carrying channelstherebetween. Note that the single-pass, cross-sectional flowconfiguration of the first liquid-cooled cold plate is also shown by wayof example only, and not by way of limitation. Both the heat transferenhancement and flow configuration through the first liquid-cooled heatsink can be varied without departing from the scope of the presentinvention. The lid structure 900 includes a coolant inlet 902 and acoolant outlet 903 which facilitate the flow of liquid coolant throughthe first liquid-cooled heat sink. Additionally, base structure 910 hasprojecting therefrom the fluid couplers 711, 712 discussed above. Thesefluid couplers comprise, in one example, rigid, hollow, cylindricalextensions from the first liquid-cooled heat sink, sized and configuredto project into corresponding receiving coolant ports in the secondliquid-cooled heat sink described above in connection with FIGS. 7A-8B.Fluid-tight coupling is facilitated by providing one or more sealingrings 905, such as O-rings, disposed about the periphery of fluidcouplers 711, 712 and/or within the receiving ports of the secondliquid-cooled heat sink. The lid structure 900 is configured withrecesses 901, 908 to facilitate the desired coolant flow path throughthe first liquid-cooled heat sink, the fluid couplers and secondliquid-cooled heat sink.

FIG. 9D is a top plan view illustrating one example of a coolant flowpath through the fluidically coupled first and second liquid-cooled heatsinks 710, 720. As described above, when the cooling apparatus isassembled onto the electronic assembly so that the second liquid-cooledheat sink is, for instance, fixedly-secured to the electronic assembly,and the separably-coupled, first liquid-cooled heat sink makes thermalcontact with the global heat spreaders to cool the optics modules, thefirst liquid-cooled heat sink also fluidically connects to the secondliquid-cooled heat sink via the illustrated fluid couplers. The fluidcouplers are, in one instance, hollow cylindrical or tubular appendages,which fit concentrically within the respective coolant ports in thesecond liquid-cooled heat sink. An elastomeric O-ring provides anannular seal when the connection is made. A single O-ring is illustratedin the embodiments depicted, however, it should be understood thatmultiple O-rings could be implemented for redundancy. Also, note thatthe fluid couplers could be integral with the base structure of thefirst liquid-cooled heat sink, or could be brazed to the base structure,along with the lid.

As noted, FIG. 9D illustrates the assembled cooling apparatus andelectronic assembly, with the flow path through the cooling apparatusdepicted by arrows and coolant flow directions 920-926, as one example.Liquid coolant flow enters 920 the cooling apparatus at the coolantinlet 902 and flows 921 through the plurality of coolant-carryingchannels of the first liquid-cooled heat sink. As noted above, theplurality of coolant-carrying first channels of the first liquid-cooledheat sink may be formed by a plurality of fins within the first heatsink. In this manner, one grouping of optics modules, and the othergrouping of optics modules, are cooled by the first liquid-cooled heatsink. Coolant flow is then directed 922 to one of the fluid couplers toflow downwards 923 into the second liquid-cooled heat sink 720 that isin thermal communication with the one or more second electroniccomponents, such as the hub chip. Coolant flow then passes 924 throughthe one or more second channels of the second liquid-cooled heat sinkbefore exiting 925 back into the first liquid-cooled heat sink to flow926 to the coolant return 903. Note that in this embodiment, the liquidcoolant flow through the second coolant-carrying heat sink isapproximately orthogonal to the coolant flow through the one or morecoolant-carrying first channels of the first liquid-cooled heat sink.This is by way of illustration only, and not limitation.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprise” (andany form of comprise, such as “comprises” and “comprising”), “have” (andany form of have, such as “has” and “having”), “include” (and any formof include, such as “includes” and “including”), and “contain” (and anyform contain, such as “contains” and “containing”) are open-endedlinking verbs. As a result, a method or device that “comprises”, “has”,“includes” or “contains” one or more steps or elements possesses thoseone or more steps or elements, but is not limited to possessing onlythose one or more steps or elements. Likewise, a step of a method or anelement of a device that “comprises”, “has”, “includes” or “contains”one or more features possesses those one or more features, but is notlimited to possessing only those one or more features. Furthermore, adevice or structure that is configured in a certain way is configured inat least that way, but may also be configured in ways that are notlisted.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below, if any, areintended to include any structure, material, or act for performing thefunction in combination with other claimed elements as specificallyclaimed. The description of the present invention has been presented forpurposes of illustration and description, but is not intended to beexhaustive or limited to the invention in the form disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the invention.The embodiment was chosen and described in order to best explain theprinciples of one or more aspects of the invention and the practicalapplication, and to enable others of ordinary skill in the art tounderstand one or more aspects of the invention for various embodimentswith various modifications as are suited to the particular usecontemplated.

What is claimed is:
 1. A cooling apparatus comprising: a firstliquid-cooled heat sink comprising at least one coolant-carrying firstchannel, the first liquid-cooled heat sink being separably coupled to anelectronic assembly comprising at least one first electronic componentand at least one second electronic component, and facilitating removalof heat generated by at the least one first electronic component; asecond liquid-cooled heat sink comprising at least one coolant-carryingsecond channel, the second liquid-cooled heat sink being fixedly securedto the electronic assembly, and facilitating removal of heat generatedby the at least one second electronic component; and fluid couplersfluidically coupling the first and second liquid-cooled heat sinks andfacilitating liquid coolant flow through the at least onecoolant-carrying second channel of the fixedly-secured, secondliquid-cooled heat sink from the separably-coupled, first liquid-cooledheat sink.
 2. The cooling apparatus of claim 1, wherein the fluidcouplers project from one of the first liquid-cooled heat sink or thesecond liquid-cooled heat sink, and project into respective coolantports in the other of the first liquid-cooled heat sink or secondliquid-cooled heat sink.
 3. The cooling apparatus of claim 2, furthercomprising a loading mechanism, the loading mechanism applying, at leastin part, a compressive loading to the first and second liquid-cooledheat sinks, with the separably-coupled, first liquid-cooled heat sinkdisposed over the fixedly-secured, second liquid-cooled heat sink, thecompressive loading facilitating forming a fluid-tight seal between thefirst and second liquid-cooled heat sinks about the fluid couplers. 4.The cooling apparatus of claim 3, further comprising sealing ringsassociated with at least one of the fluid couplers or the coolant ports,the sealing rings facilitating fluid-tight coupling between the firstliquid-cooled heat sink and the second liquid-cooled heat sink, with thecompressive loading of the first and second liquid-cooled heat sinks,and the fluid couplers disposed within the respective coolant ports. 5.The cooling apparatus of claim 1, wherein the liquid coolant flowsthrough the at least one coolant-carrying second channel of thefixedly-secured, second liquid-cooled heat sink after passing throughthe at least one coolant-carrying first channel of theseparably-coupled, first liquid-cooled heat sink.
 6. The coolingapparatus of claim 5, wherein the first liquid-cooled heat sinkfacilitates removal of heat generated by multiple first electroniccomponents, and wherein the at least one second electronic componentresides in between first electronic components of the multiple firstelectronic components, and the second liquid-cooled heat sink isdisposed over the at least one second electronic component and inthermal communication therewith.
 7. The cooling apparatus of claim 6,wherein the multiple first electronic components comprise multipleoptical modules of a hub module assembly, and the at least one secondelectronic component comprises a hub chip of the hub module assembly,and wherein the first liquid-cooled heat sink extends at least partiallyover the multiple optical modules to facilitate removal of heattherefrom, and the second liquid-cooled heat sink engages the hub chipand is disposed between the hub chip and the first liquid-cooled heatsink.
 8. The cooling apparatus of claim 7, further comprising a firstglobal optics heat spreader and a second global optics heat spreaderdisposed over respective optical modules on opposite sides of the hubchip, the first and second global optics heat spreaders facilitatingtransfer of heat from the multiple optical modules to the firstliquid-cooled heat sink.
 9. The cooling apparatus of claim 1, whereinthe separably-coupled, first liquid-cooled heat sink comprises aplurality of coolant-carrying first channels arrayed in parallel, andthe fixedly-secured, second liquid-cooled heat sink comprises aplurality of coolant-carrying second channels arrayed in parallel. 10.The cooling apparatus of claim 9, wherein the plurality ofcoolant-carrying first channels of the separately-coupled, firstliquid-cooled heat sink are oriented orthogonal to the plurality ofcoolant-carrying second channels of the fixedly-secured, secondliquid-cooled heat sink.
 11. A cooled electronic system comprising: anelectronic assembly, the electronic assembly comprising at least onefirst electronic component and at least one second electronic component;and a cooling apparatus coupled to the electronic assembly, the coolingapparatus comprising: a first liquid-cooled heat sink comprising atleast one coolant-carrying first channel, the first liquid-cooled heatsink being separably coupled to the electronic assembly and facilitatingremoval of heat generated by the at least one first electroniccomponent; a second liquid-cooled heat sink comprising at least onecoolant-carrying second channel, the second liquid-cooled heat sinkbeing fixedly secured to the electronic assembly and facilitatingremoval of heat generated by the at least one second electroniccomponent; and fluid couplers fluidically coupling the first and secondliquid-cooled heat sinks and facilitating liquid coolant flow throughthe at least one coolant-carrying second channel of the fixedly-secured,second liquid-cooled heat sink from the separably-coupled, firstliquid-cooled heat sink.
 12. The cooled electronic system of claim 11,wherein the fluid couplers project from one of the first liquid-cooledheat sink or the second liquid-cooled heat sink, and project intorespective coolant ports in the other of the first liquid-cooled heatsink or second liquid-cooled heat sink.
 13. The cooled electronic systemof claim 12, further comprising a loading mechanism, the loadingmechanism applying, at least in part, a compressive loading to the firstand second liquid-cooled heat sinks, with the separably-coupled, firstliquid-cooled heat sink disposed over the fixedly-secured, secondliquid-cooled heat sink, the compressive loading facilitating forming afluid-tight seal between the first and second liquid-cooled heat sinksabout the fluid couplers.
 14. The cooled electronic system of claim 13,further comprising sealing rings associated with at least one of thefluid couplers or the coolant ports, the sealing rings facilitatingfluid-tight coupling between the first liquid-cooled heat sink and thesecond liquid-cooled heat sink, with the compressive loading of thefirst and second liquid-cooled heat sinks, and the fluid couplersdisposed within the respective coolant ports.
 15. The cooled electronicsystem of claim 11, wherein the liquid coolant flows through the atleast one coolant-carrying second channel of the fixedly-secured, secondliquid-cooled heat sink after passing through the at least onecoolant-carrying first channel of the separably-coupled, firstliquid-cooled heat sink.
 16. The cooled electronic system of claim 15,wherein the first liquid-cooled heat sink facilitates removal of heatgenerated by multiple first electronic components, and wherein the atleast one second electronic component resides in between firstelectronic components of the multiple first electronic components, andthe second liquid-cooled heat sink is disposed over the at least onesecond electronic component and in thermal communication therewith. 17.The cooled electronic system of claim 16, wherein the multiple firstelectronic components comprise multiple optical modules of a hub moduleassembly, and the at least one second electronic component comprises ahub chip of the hub module assembly, and wherein the first liquid-cooledheat sink extends at least partially over the multiple optical modulesto facilitate removal of heat therefrom, and the second liquid-cooledheat sink engages the hub chip and is disposed between the hub chip andthe first liquid-cooled heat sink.
 18. A method comprising: fabricatinga cooling apparatus to facilitate cooling at least one first electroniccomponent and at least one second electronic component of an electronicassembly, the fabricating comprising: separably coupling a firstliquid-cooled heat sink to the electronic assembly to facilitate removalof heat generated by the at least one first electronic component, thefirst liquid-cooled heat sink comprising at least one coolant-carryingfirst channel; fixedly securing a second liquid-cooled heat sink to theelectronic assembly to facilitate removal of heat generated by the atleast one second electronic component, the second liquid-cooled heatsink comprising at least one coolant-carrying second channel; andproviding fluid couplers fluidically coupling the first and secondliquid-cooled heat sinks, the fluidic coupling facilitating liquidcoolant flow through the at least one coolant-carrying second channel ofthe fixedly-secured, second liquid-cooled heat sink from theseparably-coupled, first liquid-cooled heat sink.
 19. The method ofclaim 18, further comprising providing the fluid couplers integral withand projecting from one of the first liquid-cooled heat sink or thesecond liquid-cooled heat sink, and providing respective coolant portsin the other of the first liquid-cooled heat sink or secondliquid-cooled heat sink, and further comprising operatively disposingthe first liquid-cooled heat sink, at least partially, over the secondliquid-cooled heat sink so that the fluid couplers engage the respectivecoolant ports and fluidically couple the first and second liquid-cooledheat sinks
 20. The method of claim 19, further comprising providing acompressive loading to the first and second liquid-cooled heat sinkswith the separately-coupled, first liquid-cooled heat sink disposed overthe fixedly-secured, second liquid-cooled heat sink, the compressiveloading facilitate forming a fluid-tight seal between the first andsecond liquid-cooled heat sinks about the fluid couplers.